Large language models (LLMs) can store a vast amount of world knowledge, often extractable via question-answering (e.g., "What is Abraham Lincoln's birthday?"). However, do they answer such questions based on exposure to similar questions during training (i.e., cheating), or by genuinely learning to extract knowledge from sources like Wikipedia? In this paper, we investigate this issue using a controlled biography dataset. We find a strong correlation between the model's ability to extract knowledge and various diversity measures of the training data. $\textbf{Essentially}$, for knowledge to be reliably extracted, it must be sufficiently augmented (e.g., through paraphrasing, sentence shuffling) $\textit{during pretraining}$. Without such augmentation, knowledge may be memorized but not extractable, leading to 0% accuracy, regardless of subsequent instruction fine-tuning. To understand why this occurs, we employ (nearly) linear probing to demonstrate a strong connection between the observed correlation and how the model internally encodes knowledge -- whether it is linearly encoded in the hidden embeddings of entity names or distributed across other token embeddings in the training text. This paper provides $\textbf{several key recommendations for LLM pretraining in the industry}$: (1) rewrite the pretraining data -- using small, auxiliary models -- to provide knowledge augmentation, and (2) incorporate more instruction-finetuning data into the pretraining stage before it becomes too late.

Agent-based models (ABMs) simulate the formation and evolution of social processes at a fundamental level by decoupling agent behavior from global observations. In the case where ABM networks evolve over time as a result of (or in conjunction with) agent states, there is a need for understanding the relationship between the dynamic processes and network structure. Social networks provide a natural set of tools for understanding the emergent relationships of these systems. This work examines the utility of a collection of network comparison methods for the purpose of tracking network changes in an ABM over time or between model parameters. Among the techniques examined is a novel graph pseudometric based on heat content asymptotics, which have been shown to distinguish many isospectral graphs which are not isomorphic. Additionally, we establish the use of observations about real-world networks from network science (e.g. fat-tailed degree distribution, small-world property) for ABM validation in the case where empirical population data is unavailable. These methods are all demonstrated on systematic perturbations of an original model simulating the formation of friendships in a population of 20,000 agents in Cincinnati, OH.

We innovate the way brands are built. That means we do things differently so they're better than before. In this way, we make our clients' most important marketing assets--their brands--win in a changing world. Dentsu International is a modern marketing solutions company. Our mission is to help clients navigate, progress, and thrive in a world of change.

In multistage manufacturing systems, modeling multiple quality indices based on the process sensing variables is important. However, the classic modeling technique predicts each quality variable one at a time, which fails to consider the correlation within or between stages. We propose a deep multistage multi-task learning framework to jointly predict all output sensing variables in a unified end-to-end learning framework according to the sequential system architecture in the MMS. Our numerical studies and real case study have shown that the new model has a superior performance compared to many benchmark methods as well as great interpretability through developed variable selection techniques.

The hybrid two-seat helicopter SureFly from the electric truck company Workhorse made it off the ground this week. While a few feet of hover might seem insignificant, the passenger drone startup is hailing the untethered lift-off with a pilot outside of Cincinnati, Ohio, as a huge success. For the hybrid gas- and battery-powered vertical take-off and landing vehicle (known as VTOL), this means the copter is on its way to flying with passengers inside. Once airborne, the craft will have a 75-mile range. "People want to have something in their garage to take out and fly," SureFly CEO Steve Burns said in a call Friday afternoon.

We are on the cusp of the fourth industrial revolution, also known as Industry 4.0. With Cincinnati, Ohio declaring itself an "Industry 4.0 demonstration city," the question isn't if Industry 4.0 is coming, but rather how quickly. Technologies such as the Internet of Things (IoT), robotics and automation have ushered in this fourth age of widespread change. These technologies, along with Big Data and analytics, and cyber-physical systems sit at the heart of the Industry 4.0 model and they all run on the cloud. Over the next decade, these emerging technologies will create new real-time connections between machines, production processes, and systems that will revolutionize the way we do business.

Hippocampal dentate granule cells are among the few neuronal cell types generated throughout adult life in mammals. In the normal brain, new granule cells are generated from progenitors in the subgranular zone and integrate in a typical fashion. During the development of epilepsy, granule cell integration is profoundly altered. The new cells migrate to ectopic locations and develop misoriented basal dendrites. Although it has been established that these abnormal cells are newly generated, it is not known whether they arise ubiquitously throughout the progenitor cell pool or are derived from a smaller number of bad actor progenitors. To explore this question, we conducted a clonal analysis study in mice expressing the Brainbow fluorescent protein reporter construct in dentate granule cell progenitors. Mice were examined 2 months after pilocarpine-induced status epilepticus, a treatment that leads to the development of epilepsy. Brain sections were rendered translucent so that entire hippocampi could be reconstructed and all fluorescently labeled cells identified. Our findings reveal that a small number of progenitors produce the majority of ectopic cells following status epilepticus, indicating that either the affected progenitors or their local microenvironments have become pathological. By contrast, granule cells with basal dendrites were equally distributed among clonal groups. This indicates that these progenitors can produce normal cells and suggests that global factors sporadically disrupt the dendritic development of some new cells. Together, these findings strongly predict that distinct mechanisms regulate different aspects